Validation study of case-identifying algorithms for severe hypoglycemia using hospital administrative data in Japan

Objective The purpose of this study was to evaluate the performance of algorithms for identifying cases of severe hypoglycemia in Japanese hospital administrative data. Methods This was a multicenter, retrospective, observational study conducted at 3 acute-care hospitals in Japan. The study population included patients aged ≥18 years with diabetes who had an outpatient visit or hospital admission for possible hypoglycemia. Possible cases of severe hypoglycemia were identified using health insurance claims data and Diagnosis Procedure Combination data. Sixty-one algorithms using combinations of diagnostic codes and prescription of high concentration (≥20% mass/volume) injectable glucose were used to define severe hypoglycemia. Independent manual chart reviews by 2 physicians at each hospital were used as the reference standard. Algorithm validity was evaluated using standard performance metrics. Results In total, 336 possible cases of severe hypoglycemia were identified, and 260 were consecutively sampled for validation. The best performing algorithms included 6 algorithms that had sensitivity ≥0.75, and 6 algorithms that had positive predictive values ≥0.75 with sensitivity ≥0.30. The best-performing algorithm with sensitivity ≥0.75 included any diagnoses for possible hypoglycemia or prescription of high-concentration glucose but excluded suspected diagnoses (sensitivity: 0.986 [95% confidence interval 0.959–1.013]; positive predictive value: 0.345 [0.280–0.410]). Restricting the algorithm definition to those with both a diagnosis of possible hypoglycemia and a prescription of high-concentration glucose improved the performance of the algorithm to correctly classify cases as severe hypoglycemia but lowered sensitivity (sensitivity: 0.375 [0.263–0.487]; positive predictive value: 0.771 [0.632–0.911]). Conclusion The case-identifying algorithms in this study showed moderate positive predictive value and sensitivity for identification of severe hypoglycemia in Japanese healthcare data and can be employed by future pharmacoepidemiological studies using Japanese hospital administrative databases.


Results
In total, 336 possible cases of severe hypoglycemia were identified, and 260 were consecutively sampled for validation. The best performing algorithms included 6 algorithms that had sensitivity �0.75, and 6 algorithms that had positive predictive values �0.75 with sensitivity

Introduction
The current guidelines of the American Diabetes Association define severe hypoglycemia as an event characterized by altered mental and/or physical functioning that requires assistance from another person for recovery [1]. Although there are several causes of hypoglycemia, the majority of cases among patients with diabetes arise from use of glucose-lowering therapies, particularly insulins and sulfonylureas [2,3]. Severe hypoglycemia is a serious complication of diabetic therapy, associated with increased risk of mortality and morbidity [4][5][6][7][8] and significant socioeconomic costs, including the frequent requirement for emergency department visits and hospitalization [9].
Real-world healthcare databases are increasingly being used in pharmacoepidemiological and post-marketing surveillance studies as they contain data from large, heterogeneous populations of patients which can be queried with coded algorithms designed to identify specific populations, such as patients with certain medical conditions, characteristics, or outcomes [10]. Use of electronic databases can offer time-and cost-saving advantages over conventional methods and has wide applicability, including the assessment of real-world drug safety and effectiveness, costs, resource utilization, and treatment patterns [10][11][12]. However, the algorithms used to interrogate administrative databases are based on diagnostic codes, and the codes and databases were not designed for this purpose. As such, validation studies are needed to evaluate the performance of algorithms to correctly detect cases and outcomes of interest in the target database [10][11][12].
The accuracy of case-identifying algorithms in real-world databases can be assessed by comparing algorithm performance against a reference standard, which is the best available method for detecting the condition of interest [13]. Several prior studies have assessed the performance of algorithms that identify cases of hypoglycemia requiring in-and/or outpatient visits to North American hospitals [14][15][16][17][18], but similar validation studies have not been conducted in Japan. Prior studies have indicated racial or ethnic differences in hypoglycemiarelated emergency department visits and hospitalizations [19]. In addition, country-and health data system-specific factors have been found to limit the generalizability of validation study results across different diseases [20][21][22][23]. For these reasons, there is a need to develop and assess high-performing algorithms within the Japanese healthcare context. The primary objective of this study was to evaluate the validity of case-identifying algorithms for severe hypoglycemia using administrative data at 3 acute-care hospitals in Japan.

Study overview
This was a multicenter, retrospective observational study that investigated the validity of caseidentifying algorithms for the detection of severe hypoglycemia using Japanese hospital administrative data. The algorithms were based on diagnosis or prescription for possible hypoglycemia, defined using the International Statistical Classification of Diseases and Related Health Problems, 10th Revision (ICD-10) [24]. As ICD-10 codes do not include a code specific for "severe hypoglycemia", the current study examined all possible hypoglycemic events that required inpatient or outpatient visits to participating hospitals to identify cases of likely severe hypoglycemia.
An overview of the study design is shown in Fig 1. Eli Lilly Japan K.K. commissioned the Medical Data Vision Co., Ltd, (MDV; Tokyo, Japan) to identify candidate hospitals using the commercially available MDV database, an anonymized database containing data from health insurance claims and discharge summaries from acute-care hospitals under the Diagnosis Procedure Combination (DPC) system in Japan [25]. All possible cases of severe hypoglycemia were identified at 3 participating hospitals using a broad case-identifying algorithm based on health insurance claims data and DPC data stored at the hospitals. Following case selection at all participating hospitals, possible cases were subsequently sampled in consecutive fashion from newest to oldest for medical record review, which was conducted independently by 2 raters at each participating hospital based on review of medical record information. Raters were physicians who had experience in emergency department or inpatient diabetes treatment. Raters reviewed non-anonymized hospital record data but were blinded to health insurance claims data, DPC data, and algorithmic results. The anonymized results of medical record review were provided to MDV only. MDV calculated the positive predictive value (PPV) and sensitivity for each algorithm and provided the results of statistical analyses to Eli Lilly K.K. and Real World Data Co., Ltd, (Kyoto, Japan) for analysis and interpretation. Eli Lilly K.K. calculated additional performance metrics (prevalence, probability+, and probability-) using the results of the statistical analyses provided by MDV. Authors did not receive any patient-level data. Data collection at the participating hospitals was conducted from May 20 to June 24, 2022. This study was approved by an independent central ethics committee (MINS Research Ethics Committee, Tokyo, Japan; Application ID#220202) and conducted in accordance with the Declaration of Helsinki and the Japanese Ethical Guidelines for Medical and Biological Research Involving Human Subjects [26]. As this was a non-interventional study using anonymized patient data, the requirement for informed consent was waived by the ethical review board. Participating hospitals provided the opportunity for patients to refuse the use of their health information.

Hospital selection and study population
Hospitals with the most patients recorded in the MDV database meeting the criteria for the study population were identified, and those that had previously participated in validation studies were contacted by MDV to participate in this study. The 3 hospitals that agreed to participate were included in this study. Per agreement between MDV and the hospitals to maintain their anonymity, participating hospitals have been arbitrarily numbered herein as 1, 2, and 3. The representativeness of these hospitals was assessed by comparing the age distribution and the inpatient/outpatient ratio of the potential patients in the participating hospitals to those of potential patients in all medical institutions in the MDV database.
Eligible patients were �18 years at the date of an outpatient visit or hospital admission with possible hypoglycemia (the index date) with a diagnosis code of diabetes based on DPC or health claims data within 12 months from the index month and a diagnosis or prescription for possible hypoglycemia at the index date. The definition of possible hypoglycemia was based on the following ICD-10 diagnostic codes [24]: E10.0 (type 1 diabetes mellitus with coma;), E11.0 (type 2 diabetes mellitus with coma), E14.0 (unspecified diabetes mellitus with coma), E15

PLOS ONE
Validation of case-identifying algorithms for severe hypoglycemia using real-world hospital data (non-diabetic hypoglycemic coma), E16.0 (drug-induced hypoglycemia without coma), E16.1 (other hypoglycemia), and E16.2 (hypoglycemia, unspecified) or a high-concentration (�20% mass/volume) injectable glucose prescription for possible hypoglycemia. Patients with diagnoses or prescriptions recorded after the hospital admission date were not included in order to exclude hypoglycemic events arising secondary to treatment while in hospital.
The eligibility evaluation period was defined in the study protocol as the earliest time point at which data could be obtained to July 31, 2021. During the study, a practical measure was taken to avoid any excessive imbalance in the numbers of patients assessed at the 3 hospitals, wherein the start of the evaluation period was determined for each hospital based on the number of patients who could potentially meet the eligibility criteria at the hospital during the evaluation period. The actual start dates for eligibility evaluation across the 3 hospitals were June 1, 2018; January 1, 2019; and January 1, 2020 (dates are provided in no particular order, i.e., numbering of hospitals does not correspond to the order or timing of the eligibility evaluation periods). For patients who met the above criteria 2 or more times, only the most recent episode was used, and the previous hypoglycemic episodes were excluded.

Medical record review
For the reference standard, all possible eligible cases were assessed independently by 2 physicians at each participating hospital who had experience in the emergency department or in diabetes treatment. Assessments of medical record information were conducted using a judgement sheet based on the American Diabetes Association definition of severe hypoglycemia (S1 Table). In brief, the criteria were 1) the presence of sympathetic symptoms (finger tremor, palpitation, cold sweat, etc) with no recovery without assistance by family members or medical treatment; 2) the presence of central nervous system symptoms (decreased level of consciousness, abnormal behavior, convulsions, coma, etc) with no recovery without assistance by family members or medical treatment; and 3) interventions for recovery provided by family members or medical institutions (oral intake of drinking water containing glucose, injection of glucagon, intravenous injection of high-concentration [�20% mass/volume] glucose, etc). A case was judged as severe hypoglycemia if criterion 3 was met with either criterion 1 or 2. If the judgements of the 2 assessors did not match, a third assessor with more experience in treating severe hypoglycemia determined whether the case was severe hypoglycemia. All assessors were blinded from viewing health insurance claims data, DPC data, and algorithmic results. The assessment window for severe hypoglycemia was the period within ±30 days of the index date.
The assessors also evaluated additional patient information in hospital medical records, including blood glucose levels at the time of the event (date, continuous variable/unclear). If multiple glucose values were available for a patient, the minimum value between the event occurrence and the hospital visit was recorded. Additionally, assessors determined whether patients were under sulfonylurea or insulin treatment (no/yes/unclear) at the time of the event, as these glucose-lowering medications have previously been associated with the highest risk for hypoglycemic-related hospital visits [19]. The assessment window for these data was from 30 days before the index date to the index date.

Algorithms
We identified cases of possible severe hypoglycemia in hospital administrative data by the application of 61 case-identifying algorithms that were based on a combination of disease names for hypoglycemia (ICD-10 codes E10.0, E11.0, E14.0, E15, E16.0, E16.1, and E16.2) and/or prescription of high-concentration (�20% mass/volume) injectable glucose. The algorithms used herein are listed in S2 Table. ICD-10 diagnostic codes and health insurance drug codes used in the study are listed in full in S3 and S4 Tables, respectively.

Sample size
Sample size was determined in accordance with a Japanese regulatory notice which states that validation studies plan for at least 100 cases with the target condition within all possible cases [27]. This study planned to consecutively sample 260 patients, on the assumption that at least 100 patients who met the inclusion criteria would be judged to have cases of severe hypoglycemia by medical record review. This assumption was based on a prior study by Min et al. (2019) that found 12 of 15 inpatients with possible hypoglycemia (80%) and 44 of 103 emergency outpatients with possible hypoglycemia (43%) were cases of severe hypoglycemia [17]. A preliminary survey of patients who met the inclusion criteria in the current study showed that there were 62 inpatients and 198 outpatients, which, if the proportions derived from the Min et al. study [17] were applied to the current study, would result in the identification of approximately 50 cases of severe hypoglycemia among the inpatients (i.e., 80% sensitivity) and 85 cases among the outpatients (i.e., 43% sensitivity), or 135 overall.

Statistical analyses
Selected case information from hospital health insurance claims and DPC data were descriptively summarized, including patient sex, age and age group (<65 years or �65 years) at the index date, classification as inpatient or outpatient (including emergency and other outpatients) at the index date, and the Sundararajan version of the Charlson score [28]. For continuous variables, mean with standard deviation or median with interquartile range (IQR) were calculated. For categorical variables, frequency and percentage were summarized.
Case terminology was based on the Standards for Reporting of Diagnostic Accuracy Studies guidelines [29], in which cases with the presence of severe hypoglycemia were termed "cases with the target condition" and cases identified by algorithm (test under evaluation) were termed "index test-positive cases". Cases were considered: (a) "true positives" if identified as "true" by both the algorithm and medical record review; (b) "false negatives" if identified by medical record review but not by the algorithm; (c) "false positives" if identified by the algorithm but not by medical record review; and (d) "true negatives" if identified as false by both the algorithm and medical record review. In the medical record review, inter-rater agreement was assessed via a Kappa statistic.
As the regulatory agency in Japan recommends PPV and sensitivity as measures of validation for rare outcome events [27], the focus of our validation analysis was on these metrics. Normal approximation to the binomial distribution was used to calculate 95% confidence intervals (CIs) for PPV and sensitivity. We also report prevalence, probability+, and probability-. S1 Fig shows the calculations for the performance metrics evaluated in this study. PPV was calculated by dividing the number of true-positive cases by the number of index test-positive cases (the sum of the true-positive and false-positive cases). Due to the impractical nature of reviewing all medical records in a database when the prevalence of the target condition is low, for the evaluation of sensitivity, we assumed that all cases with the target condition met the eligibility criteria and were included in the study. Sensitivity was calculated by dividing the number of true-positive cases by the number of cases with the target condition (sum of truepositive and false-negative cases). Prevalence was calculated by dividing the number of cases identified as hypoglycemia by medical record review (sum of the true-positive and false-negative cases) by the total number of cases. Probability+ was calculated by dividing the number of index test-positive cases (sum of the true-positive and false-positive cases) by the total number of cases. Probability-was calculated by dividing the number of index-case negative cases (sum of the true-negative and false-negative cases) by the total number of cases.
Performance metrics were also determined for subgroups, including sex (male or female), age group (<65 years old or �65 years), and hospitalization classification (inpatient or outpatient). In these analyses, patients were excluded from the relevant subgroup analysis if values were missing for the variable used for subgrouping.

Analysis population
The representativeness of the selected medical institutions is shown in S5 Table, which compares the age distribution and inpatient/outpatient ratio of the patients in the 3 selected hospitals to those of potential patients in all medical institutions in the MDV database. The total number of possible cases of hypoglycemia identified by algorithm was 336, of which 260 were sampled for validation. Of these, 72 were identified as cases of severe hypoglycemia by medical record review (Fig 2). The Kappa statistic was 1, indicating 100% agreement between the 2 medical record reviewers at each hospital.

Assessment of algorithms
Algorithms 5, 6, 17, 18, 59, and 60 were considered PPV-focused algorithms and included only cases that had both a diagnosis of hypoglycemia and prescription of high-concentration glucose, which improved the performance of the algorithms to correctly classify algorithmpositive cases as severe hypoglycemia but reduced the number of true-positive cases identified, thereby reducing sensitivity. Algorithm 5 had the highest PPV and sensitivity (sensitivity: 0.389; 95% CI 0.276, 0.501; PPV: 0.778; 95% CI 0.642, 0.914). Algorithm 6, which was identical to Algorithm 5 but excluded suspected diagnoses, had slightly lower sensitivity and PPV (sensitivity: 0.375; 95% CI 0.263, 0.487; PPV: 0.771; 95% CI 0.632, 0.911). Restricting the variety of diagnoses considered by Algorithm 5 and 6 further (i.e., Algorithms 17, 18, 59, and 60) slightly reduced the number of true-positive cases identified and did not improve the PPV of the algorithms (Table 2).

PLOS ONE
Validation of case-identifying algorithms for severe hypoglycemia using real-world hospital data Algorithms 4 and 58 were considered balanced-type algorithms. The best performing balanced-type algorithm was Algorithm 4, which included any diagnosis of hypoglycemia except suspected diagnoses, but did not include drug prescription as a second condition of identification. This substantially reduced both the PPV and sensitivity of the algorithm (PPV: 0.500; 95% CI 0.399, 0.601; sensitivity: 0.653; 95% CI 0.543, 0.763) compared to the best-performing PPV-and sensitivity-focused algorithms ( Table 2). Algorithm 58 restricted diagnoses considered by Algorithm 4 to only E16.2-coded diagnoses ("hypoglycemia, unspecified"), which decreased the number of true-positive cases identified by 8 cases, reducing sensitivity (0.542; 95% CI 0.427, 0.657) but not PPV (0.520; 95% CI 0.407, 0.633).
The sensitivity and PPV of 14 selected, best-performing algorithms (sensitivity-focused, PPV-focused, and balanced-type) are shown by subgroup in S7 Table. For the subgroup analyses, the number of cases sampled for validation included 158 males and 102 females in the sex subgroup, 70 patients <65 years and 190 patients �65 years in the age subgroup, and 124 inpatients and 136 outpatients in the hospitalization subgroup. No major differences were observed between the sensitivity or PPV of selected algorithms in any of the subgroups compared to those in the overall population. In an informal comparison of the data within the hospitalization subgroup, PPVs for the inpatient group trended approximately 10% higher across the PPV-focused algorithms compared to PPVs for the outpatient group (S7 Table).

PLOS ONE
Validation of case-identifying algorithms for severe hypoglycemia using real-world hospital data

PLOS ONE
Validation of case-identifying algorithms for severe hypoglycemia using real-world hospital data

Discussion
This study evaluated the performance of 61 case-identifying algorithms for the detection of severe hypoglycemic events requiring inpatient or outpatient hospital visits. None of the algorithms had both high PPV and high sensitivity for the detection of cases judged to have the target condition by medical record review. Six algorithms were identified that had sensitivity �0.75, and 6 algorithms were identified that had a PPV �0.75 with sensitivity �0.30. Two algorithms with moderate PPV and sensitivity (�0.50 each) were also identified. Algorithms 1 and 2 (any hypoglycemia diagnosis or prescription of high-concentration glucose, including and excluding suspected cases, respectively) were the best performing sensitivity-focused algorithms whereas Algorithms 5 and 6 (any hypoglycemia diagnosis plus prescription of highconcentration glucose, including and excluding suspected cases, respectively) were considered the best performing PPV-focused algorithms. In the Japanese health claims system, a suspected disease flag is generally included to support claims for diagnostic tests, and the flag is removed if a diagnosis of exclusion for the disease is confirmed. Therefore, Algorithms 2 and 6 may be preferrable for most applications, as these excluded suspected cases.
Prior studies on the validation of case-finding algorithms for hypoglycemia using electronic healthcare databases reported PPVs ranging from 48% to 99.2% [14][15][16][17][18]. Sensitivity was also reported by Hodge et al., 2017 [15], who found their algorithm had a high PPV at 94% but low sensitivity (12.7%). In contrast, in their validation study using United States Medicare claims data, Yang et al., 2022, reported moderate PPV (69.2%) and moderate sensitivity (83.9%) across all acute-care encounters, with better accuracy observed for emergency department visits (PPV: 92.9%; sensitivity: 100%) than inpatient visits (PPV: 53.7%; sensitivity: 71.0%) [18]. These data suggest that the current best-performing algorithms, Algorithms 2 and 6, may offer improved sensitivity or PPV over some of the previously proposed algorithms. However, there are a number of study design and methodological differences that must be considered when making cross-study comparisons of algorithm performance, including, for example, the use of different administrative codes and/or different versions of the ICD to define hypoglycemia [14,16,17] and different study cohort inclusion criteria (e.g., Hodge et al., 2017 [15], only assessed patients >65 years of age). In addition, ethnic-, country-, and site-specific factors should also be considered [19][20][21][22][23]. Additional studies are needed to confirm the performance of the currently described algorithms in other contexts. Our inclusion of comprehensive assessment metrics for all 61 algorithms examined in the current study should facilitate future comparisons and the refinement of algorithms.
It is well recognized that different types of case-identifying algorithms may be best suited to different applications and, when developing algorithms, certain performance metrics may need to be prioritized over others, depending on the application [30]. An algorithm with high PPV would be expected to better classify algorithm-positive cases as severe hypoglycemia whereas an algorithm with high sensitivity would be expected to identify most severe hypoglycemia cases in a patient population [30]. High-sensitivity algorithms that broadly define severe hypoglycemia may be the most appropriate for assessment of drug safety signals, as the use of broad definitions would maximize the number of cases of severe hypoglycemia identified in a patient population [10]. There are frequently trade-offs between the accuracy metrics [30-32]. As PPV represents the likelihood that an algorithm correctly identifies a case as severe hypoglycemia, prioritizing high PPV would increase the number of true-positive cases of severe hypoglycemia identified, which would be expected to reduce bias in the relative risk estimates [30]. As the algorithms with high sensitivity (�0.75) in this study did not also have high PPV (�0.75), more false-positive cases would be identified by these algorithms. Conversely, as the algorithms with high PPV defined in this study did not also have high sensitivity, some target cases may be missed when using these algorithms. When choosing among the best-performing algorithms described in this study for future studies, prioritization of sensitivity versus PPV should be assessed in terms of the intended application and setting. The balanced-type algorithms identified in this study might be of limited use for future studies using hospital administrative data in Japan, given their lower PPV and sensitivity than the best performing sensitivity-and PPV-focused algorithms.
Overall, no major differences were observed between the performance of selected algorithms in any of the subgroups compared to the overall population. In the hospitalization subgroup, PPV for the inpatient group was approximately 10% higher than that for the outpatient group. Consistent with this finding, Min et al. (2019) [17] also reported a higher PPV for "hospitalization events" than for outpatient "emergency department events" (80% and 48%, respectively). Nevertheless, the PPV for the outpatient subgroup in the current study was approximately 70% for the PPV-focused algorithms, suggesting that our algorithms may be well suited for case identification in outpatient populations.
Strengths of this study include validation of algorithms based on the findings of 2 physicians at each of 3 sites who independently reviewed medical records. In addition, the age distribution and ratio of inpatients to outpatients at the 3 institutions were similar to those in the overall MDV database, indicating that the selected medical institutions were representative of the institutions covered in the MDV database in terms of these characteristics. The patients included in the current validation study also appear representative of severe hypoglycemia cases reported more broadly within the diabetic population in Japan, based on a comparison of the patient characteristics (i.e., the ratio of males to females, ratio of inpatients to outpatients, and age ranges) in this study and those of patients reporting severe hypoglycemia in a national Japan Diabetes Society survey conducted at 193 medical facilities [33].
This study had several limitations. Notably, since the MDV database covers Japanese DPC hospitals only, the applicability of the algorithms evaluated in this study to the other settings is unknown. Performance of the algorithms in other contexts may differ and particular consideration should be given to differences in coding practices and in the clinical characteristics of the study cohort represented in different electronic databases [30]. We also calculated sensitivity under the assumption that all cases with the target condition met the eligibility criteria and were included in the study, which may not have captured all cases in the hospital databases. In addition, fewer true-positive cases were found in the <65 years category compared to the �65 years category. However, this finding is not unexpected given that the highest rates of severe hypoglycemia have previously been reported among elderly patients and those with multiple comorbidities [34]. Finally, negative predictive value and specificity were not calculated, as both statistics pertain to patients without possible hypoglycemia, which were not included in this study, thus rendering these statistics inapplicable to other settings which include patients both with and without possible hypoglycemia.

Conclusions
This study evaluated several algorithms designed to detect cases of severe hypoglycemia in Japanese hospital administrative databases that contained claims and DPC data. Our analysis identified the best performing algorithms as those that excluded suspected cases and required either a diagnosis for possible hypoglycemia or a prescription of high-concentration glucose (Algorithm 2) or both a diagnosis of possible hypoglycemia and a prescription of high-concentration glucose (Algorithm 6). It is anticipated that the high-performing case-identifying algorithms in this study will be of utility in future studies that evaluate severe hypoglycemia in Japan and utilize hospital administrative data with consideration of the suitability of the type of algorithm (e.g., high sensitivity versus high PPV), depending on the purpose and context of application. Additional studies are warranted to confirm the current findings in and other Japanese healthcare database systems.
Supporting information S1 Fig. Calculations for performance metrics. Cases of severe hypoglycemia identified by the algorithm were termed 'index test-positive' cases. Independent manual chart reviews by 2 physicians at each participating hospital were used as the reference standard. Cases were considered: (a) "true positives" if identified as severe hypoglycemia by both the algorithm and medical record review; (b) "false negatives" if identified by medical record review but not by the algorithm; (c) "false positives" if identified by the algorithm but not by medical record review; and (d) "true negatives" if identified as not cases of severe hypoglycemia by both the algorithm and medical record review. Algorithm validity was evaluated using standard performance metrics. PPV represents the likelihood that the algorithm correctly identifies a case as severe hypoglycemia. Sensitivity measures the true positive ratio (calculated herein under the assumption that all cases with the target condition met the eligibility criteria and were included in the study). Prevalence was calculated by dividing the number of cases identified as severe hypoglycemia by medical record review by the total number of cases. Probability+ was calculated by dividing the number of index test-positive cases by the total number of cases. Probability-was calculated by dividing the number of index-case negative cases by the total number of cases. NPV (the likelihood that the algorithm correctly classifies a case as not severe hypoglycemia) and specificity (the true negative ratio when comparing the performance of the algorithm to the reference standard) are not included herein as these statistics pertain to patients who do not have severe hypoglycemia. NPV and specificity derived from the current study, which did not include patients without possible hypoglycemia, would not be applicable to other settings that include patients with and without possible hypoglycemia. NPV = negative predictive value; PPV = positive predictive value. (TIF) S1